Add grails.mongodb.buildIndexes and buildIndexesAsync for MongoDB index creation on startup - #16208
Open
codeconsole wants to merge 7 commits into
Open
Add grails.mongodb.buildIndexes and buildIndexesAsync for MongoDB index creation on startup#16208codeconsole wants to merge 7 commits into
codeconsole wants to merge 7 commits into
Conversation
…rtup GORM created and reconciled every index declared in a domain class mapping block each time the datastore started, with no way to switch it off. That is unwanted when deploying against live data whose indexes are managed separately: the deployment builds indexes against production collections and reconciles the index set the running application depends on. Setting grails.mongodb.buildIndexes = false leaves the server's indexes exactly as they are - no createIndex or collMod command is issued for any domain class, including for domain classes registered after startup. Persistence and querying are unaffected and continue to use whichever indexes already exist. The setting is resolved per connection like every other connection setting, so index building can be switched off globally and left on for an individual connection.
The constructors that take a MongoClient (or a MongoClientSettings.Builder) built the default connection source from a bare MongoConnectionSourceSettings, so every setting under grails.mongodb that describes how the datastore behaves - stateless, transactional, engine, flush mode, decimalType, buildIndexes - was silently left at its default. Only databaseName survived, because it is set explicitly from the mapping context. That path is not obscure: Spring Boot's MongoDB auto-configuration contributes a MongoClient bean, and MongoDbGormAutoConfiguration hands that client to GORM whenever it is present, so a Spring Boot application configuring GORM through grails.mongodb was being ignored. The settings are now bound from the configuration on this path too. The connection details in them go unused, as the client is supplied already connected.
… thread MongoDB answers a createIndex command only once the index has been built, and the datastore builds every declared index from its constructor, so an application waits at startup for all of them in turn. On an empty collection that is instant; on a large existing collection it is minutes of a deployment spent waiting. Setting grails.mongodb.buildIndexesAsync = true hands the startup build to a single daemon thread per connection, named gorm-mongo-index-build-<connection>, and returns immediately. Indexes are still built one at a time rather than all at once against the server. Two consequences are documented rather than hidden: a query issued before its index exists is served without it (a unique index likewise constrains nothing until the build completes), and a build failure can no longer fail startup, so it is logged at error level instead. The setting covers the startup build only - a domain class registered later is still indexed on the registering thread, which keeps the tenant context that path can depend on.
A successful index build said nothing at all: only conflicts, TTL updates and
failures were logged, so there was no way to tell how much of startup went on
building indexes, and with buildIndexesAsync no signal that the background
build had finished.
The build now logs one summary when it completes:
Applied 7 index declaration(s) from 3 domain class(es) to database [myDb] in 412ms
The count is declarations applied rather than indexes built, because
createIndex is idempotent and the server accepts one that already exists
without doing any work - claiming otherwise would be wrong on every restart
after the first. A build with failures is logged at warn and reports how many.
Each index also logs its own elapsed time at debug, which is how to find the
one slow index behind a slow summary.
The tests now run against logback rather than the no-op SLF4J binding, as the
sibling data modules do, so that a test can assert on what was logged; the
root logger is quietened to WARN so this does not add noise to the build.
The previous summary counted declarations applied and said, in as many words,
that it could not tell whether any of them did work. That was wrong about the
server: createIndexes answers with numIndexesBefore, numIndexesAfter and a
"note: all indexes already exist", and it is the driver's createIndex helper
that discards the response and returns only the index name.
Rather than hand-roll the command to read that response - the mapping from
IndexOptions to an index specification is not worth reimplementing - the build
lists the indexes a collection already has, once per collection, and reports:
Index build for database [myDb] finished in 412ms: 2 created, 5 already present, from 3 domain class(es)
That split is what makes the elapsed time mean something. Measured against
MongoDB 7.0 with 200k documents: creating an index took 172ms on the calling
thread, re-issuing the same declaration 1.2ms. A restart that changed no
mappings reports everything as already present, so a summary reporting created
indexes is the one that accounts for a slow start.
The listing is lazy, so an entity declaring no indexes costs no round trip, and
it replaces the listing the conflict path used to make for itself. If it cannot
be read the build still runs and the summary falls back to reporting how many
declarations were applied.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 8.0.x #16208 +/- ##
==================================================
+ Coverage 54.1238% 54.1510% +0.0272%
- Complexity 20307 20328 +21
==================================================
Files 2107 2107
Lines 101144 101228 +84
Branches 17921 17935 +14
==================================================
+ Hits 54743 54816 +73
- Misses 38595 38602 +7
- Partials 7806 7810 +4
🚀 New features to boost your workflow:
|
…n.groovy The configuration examples were written only in the application.groovy DSL, which reads as though the Groovy form were the only one; application.yml is what a generated Grails application actually ships with. Both forms are now shown, and the settings are noted as being read by name: they are not relaxed-bound the way Spring Boot's own properties are, so a kebab-case spelling is ignored and silently leaves the default in place.
✅ All tests passed ✅🏷️ Commit: 6fbb305 Learn more about TestLens at testlens.app/docs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two settings for the MongoDB index build that runs when the datastore starts, plus the reporting to make either decision an informed one.
All of these are runtime configuration, set in
grails-app/conf/application.yml(orapplication.groovy) — nothing inbuild.gradle.grails.mongodb.buildIndexesGORM creates and reconciles every index declared in a mapping block each time the datastore starts, with no way to switch it off. That is unwanted when deploying against live data whose indexes are managed separately, by a DBA or a migration step: the deployment builds indexes against production collections and reconciles the index set the running application depends on.
With the setting off, no
createIndexorcollModis issued for any domain class and the indexes on the server are left exactly as they are. Persistence and querying are unaffected and keep using whichever indexes exist. It also covers domain classes registered after startup, and resolves per connection like every other connection setting, so index building can be switched off globally and left on for one connection:A collection that has never started up with index building enabled will have no declared indexes at all, so this is for environments where the indexes are already in place or are applied by other means. It governs only what GORM derives from mapping blocks; an explicit
createIndexin application code is unaffected.grails.mongodb.buildIndexesAsyncMongoDB answers a
createIndexonly once the index has been built, so by default startup waits for every declared index in turn. Measured against MongoDB 7.0 with 200k documents: 172ms to create one index on the calling thread, 1.2ms to re-issue the same declaration.background: truehas been ignored since MongoDB 4.2 and costs the same.With this enabled the startup build runs on one daemon thread per connection, named
gorm-mongo-index-build-<connection>, and startup continues without waiting. Indexes are still built one at a time rather than all at once against the server.Worth planning for:
uniqueindex likewise constrains nothing until the build finishes.Index build reporting
A successful build previously logged nothing at all — only conflicts, TTL updates and failures — so there was no way to tell how much of startup went on indexes, and with the async build no signal that it had finished. It now logs one summary:
The created/already-present split is what makes the elapsed time interpretable: a restart that changed no mappings reports everything as already present and costs milliseconds, so a summary reporting indexes created is the one that accounts for a slow start. A build with failures is logged at
WARNand reports how many. Each index also logs its own elapsed time atDEBUGunderorg.grails.datastore.mapping.core, which is how to find the single slow index behind a slow summary.createIndexesreportsnumIndexesBefore/numIndexesAfter, but the driver'screateIndexhelper discards the response and returns only the index name. Rather than hand-roll the command — theIndexOptionsto index-specification mapping is not worth reimplementing — the build lists the indexes a collection already has. That listing is lazy, so an entity declaring no indexes costs no round trip, and it replaces the listing the conflict path used to make for itself. If it cannot be read, the build still runs and the summary falls back to reporting how many declarations were applied.Settings were ignored when the application supplies its own
MongoClientFound while testing the above. The constructors taking a
MongoClient(or aMongoClientSettings.Builder) built the default connection source from a bareMongoConnectionSourceSettings, so everygrails.mongodbsetting describing how the datastore behaves —stateless,transactional,engine, flush mode,decimalType, and nowbuildIndexes— was silently left at its default. OnlydatabaseNamesurvived, because it is set explicitly from the mapping context.That path is not obscure: Spring Boot's MongoDB auto-configuration contributes a
MongoClientbean andMongoDbGormAutoConfigurationhands that client to GORM whenever one is present, so a Spring Boot application configuring GORM throughgrails.mongodbwas being ignored. The settings are now bound from the configuration on that path too; the connection details in them go unused, as the client arrives already connected.This is a behaviour change for anyone who had been running on defaults without knowing it — a
stateless = trueortransactional = truethat was previously discarded now takes effect.Notes
grails-data-mongodb/docsunder Querying Indexing and Advanced Configuration, with the 8.0 release notes updated. Those pages showed configuration only in theapplication.groovyDSL, so they now show theapplication.ymlform as well, and note that these settings are read by name — they are not relaxed-bound like Spring Boot's own properties, so a kebab-case spelling such asbuild-indexesis ignored and leaves the default in place.grails-data-neo4j/corealready does, so a test can assert on what was logged. The root logger is pinned toWARNby alogback-test.xml, and the build output is unchanged.